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ABSTRACT 

MediaView is a computer program that provides a generic infrastructure for authoring 
and interacting with multimedia documents. Among its many applications is the ability to 
furnish a user with a comprehensive environment for analysis and visualization. With 
MediaView the user produces a "document" that contains mathematics, datasets and 
associated visualizations. From the dataset or embedded mathematics animated 
sequences can be produced in situ. The mathematical content of the "document" can be 
explored through manipulation with Mathematica™. Since the "document" is all digital, it 
can be shared with a co-worker on a local network or mailed electronically to a colleague 
at a distant site. Animations and any other substructure of the “document” persist through 
the mailing process and can be awakened at the destination by the recipient. 


INTRODUCTION 

Once published in printed form any document, book or paper, dies a little — or at least 
ossifies. This is especially true with a scientific publication and is most pronounced in 
journals where an author is subject to stringent page count limitations. Forced to adhere 
to that, many papers become so taut as to sacrifice exposition. Figures or illustrations that 
might have helped to enlighten are omitted for the sake of brevity. Also, mathematical 
presentations are reduced to a bare minimum — clarifying intermediate steps in a 
development are often sacrificed due to space limitations. And, unless it’s a highly 
specialized publication, the likelihood of finding important fragments of computer code in 
a paper is nearly nil. In an electronic publishing environment most of these limitations 
disappear. There is an opportunity to resuscitate the work and restore to it much of the _ 
vitality that accompanied its preparation and, perhaps, presentation. | 


Armed with modern workstation technology, we can provide an electronic reading 
environment that permits one to reach into this new form of publication and pull out the 
mathematics so that we can manipulate it to experiment with it, to derive new results, to 
simulate different situations with different parameters; we can take the illustrations and 
move the eye-point, change the lighting conditions or use the model with our own 
algorithms; we can manipulate the animations and study the effects to learn and 
understand what has been accomplished; and we can make it possible for people to 
extract useful material and incorporate it with their work. 


MediaView, developed at Los Alamos National Laboratory, is the system that makes it 


possible to produce a document like this. MediaView is a computer program which 
provides a generic infrastructure for authoring and interacting with multimedia 
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documents. With MediaView, documents can be developed for professional or personal 
use, technical or administrative. Because of its emphasis on mathematics and graphics, it 
is especially useful to scientists and engineers, or others, who have a need for analysis of 
data and visualization of information. For example, one application of MediaView has 
been to produce an interactive version of papers from recent SIGGRAPH proceedings.! 
Some of the potential components of a MediaView document are shown in Figure 1. 
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Figure 1. Components of a MediaView Document 


HOW IT WORKS 

Taken individually, the components of multimedia technology appearing in MediaView 
are not new. There are already several programs which manipulate text and graphics, 
including high resolution digital color images. A few systems also provide for embedded 
voice messages, but most are experimental and require expensive add-on hardware. 
Animation too can be produced, by video add-in cards and videodisk players. Finally, 
textual annotation is even available through the Post-it" metaphor in some word 
processing systems. What is not available, however, is a completely integrated system, 
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hardware and software, that combines all of these features. This is one of the most | 
important attributes that MediaView brings to digital publishing, but there many more. 


As suggested in Figure 1, MediaView documents have the following features: 


¢ Keyword retrieval of any word or phrase in the volume and random access to that 
part of the document. Any collection of text can be copied and then pasted into a 
text editor or word processing program. 


¢ Figures that accompany text can be viewed and, if desired, extracted for processing by 
another program. This capability applies to simple drawings or 24 bit/pixel full color 
images. | 


¢ Another icon style indicates that an audio insert is available for listening, e.g. part of 
the question and answer session. The sound fragment can be played back through a 
built-in speaker or it can be captured by a sound editor for incorporation in another 
multimedia document. 


¢ The reader may wish to make notes in the "margin" of the document. These can 
be voice annotations or could be in the form of textual or graphical "stick-on" notes 
which are logically attached to a position in the document. Subsequent readers can, if 
permitted, hear and see those comments. Thus, a teacher who is using MediaView in 
a course can expand on points in a document in situ. 


ANALYSIS AND VISUALIZATION 

Some of the most important features offered by MediaView are associated with 
mathematical analysis and visualization. These are described in detail below. In reading 
of them one should bear in mind that any of the analyses or visualizations can be shared 
with a co-worker on a local network or mailed electronically to a colleague at a distant 
site, making MediaView a true interpersonal visualization tool. Animations and any other 
substructure of the "document" persist through the mailing process and can be awakened 
at the destination by the recipient. 


Linking to Mathematica 

Equations that appear in a MediaView document have a backing format that is compatible 
with the Mathematica? language. Thus, by clicking on an equation, its semantics are 
conveyed to Mathematica, where the user can perform a variety of symbolic and 
numerical operations. Figure 2 is an example of symbolic mathematics manipulation and 
graphical visualization using equations appearing in a recent SIGGRAPH paper? A 
portion of the paper is shown in the window at the left. After clicking on a Mathematica 
icon the outline format for a Mathematic notebook appears, as seen in the upper right. By 
manipulating entries in the notebook the user produced the density diagram in the lower 
left and the filter function plots to the lower right. Equations that were used in their 
production appear close by. 
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1. Introduction 
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Do{DensityPlot{Sin[x*2+y*2], {x, 0, 4Pi}, {y, -2Pi, 2P1}, 
PlotPoints->20+10 i, Mesh->False], (i, 0, 25}] 





Figure 2. Mathematical Analysis and Visualization with Mathematica 


Locally Derived Animations 

Another possibility for analysis and visualization is to perform a simulation wholly within 
Media View and use its dataset animation facility to view the result. Once the simulation 
is complete a window will open and the animation will be displayed there. Controls 
attached to the window allow single stepping and selective freezing and extraction of 
frames. Figure 3 shows an example of this using data from a paper from the 1989 
SIGGRAPH conference.’ 
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Figure 3. The MediaView Dataset Animation Facility 


Visualization of remotely-produced datasets 

MediaView also provides a facility for visualizing datasets that were produced on another 
computer, say a supercomputer that is accessible through a network connection. These 
data can be rendered to produce a single frame of output or an animated sequence. Figure 
4 is one frame from a simulation of global warming that was performed on a Cray Y/MP. 
In MediaView an entire year’s worth of data can be shown as an animation. 


IMPLEMENTATION DETAILS 

MediaView is an application for the NeXT Computer that integrates text, mathematical 
notation, illustrations, color images, animations, video segments and audio annotations, as 
well as textual and graphical annotations. The heart of the application is a custom 
multimedia object, which is based on the NeXTStep™ Text object.5 Support was added to 
accommodate arbitrary NeXTStep views. This allowed the incorporation of custom 
multimedia control and display objects into a document. Several control panels were 
constructed with the Interface Builder and a considerable amount of Objective-C code 
was written to realize the various user interface objects and to build the multimedia 
documents. While the application builds upon some multimedia techniques and packages 
already available in the NeXT system, the combination provides unique scope and 
capability for manipulating electronic documents. 
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Figure 4. Visualization of Global Warming Data Produced on a Supercomputer 


OTHER APPLICATIONS AND FUTURE PLANS 


Other uses for MediaView range from a customized multimedia newspaper to an 
interactive children’s book. In addition, MediaView can accommodate maps, CAD 
diagrams and a custom component, which can be any multimedia entity that is not 
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standard to MediaView, but is required for a particular application. Finally, anything that - 
MediaView can display can be printed or saved as a PostScript file, which is how all the 
figures included here were captured. 


While MediaView is currently available only for the NeXT computer, it can be ported to 
any platform that provides the NeXTStep programming environment. IBM recently 
announced the availability of NeXTStep for their PS/2 and RS/6000 computers. 
MediaView will be ported to both of those platforms when the software becomes 
available. 


REFERENCES 

1. Dick Phillips, Interactive SIGGRAPH Proceedings: A New Form of Publication, 
Presented at EDUCOM’89 (Ann Arbor, Mich., October 15-18, 1989). In Computer 
Graphics 24 (1), January 1990. 


2. Stephen Wolfram, Mathematica: A System for Doing Mathematics by Computer, 
Addison-Wesley, Reading, MA, 1988. 


3. Don P. Mitchell and Arun N. Netravali, Reconstruction Filters in Computer Graphics, 
Proceedings of SIGGRAPH’ 88 (Atlanta, GA, August 1-5, 1988). In Computer Graphics 
22, 4 (August 1988), 221-228. 


4. David Baraff, Analytical Methods for Dynamic Simulation of Non-penetrating Rigid 
Bodies, Proceedings of SIGGRAPH’89 (Boston, MA, July 31-August 4, 1989). In 
Computer Graphics 23, 3 (July 1989), 223-231. 


5. NeXT, Inc., The NeXT System Reference Manual, Part 2: Reference, Product #N6007, 
1989. 


Visualization °90 SUBMISSION DRAFT — NOT FOR ATTRIBUTION 7 


